The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space.

In addition, a pre-defined search grid is also necessary. With these input parameters, we firstly distribute the total allowed loss into each individual module. First, it's indeed a trade-off between the efficiency and efficacy as the speedup ratio goes higher at the cost of lower Thus, in the real application, users need to decide what's the best We could have chose another cutoff like 1 % accuracy with lower speedup ratio to report, but this won't help too much when comparing different baseline methods. D.1 LSTM result A 2-layer LSTM model is composed of two large matrices layers and one large softmax layer. Thus, despite the matrix is much smaller and well approximated by DRONE, the overall acceleration on GPU is less.

The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space.

The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space.